Papers with coherence evaluation
SNaC: Coherence Error Detection for Narrative Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | SNaC framework is used to evaluate long summaries, but it fails to identify gaps in coherence . nallapati and colleagues have developed a framework for fine-grained annotations of long summarizations . |
| Approach: | They propose a narrative coherence evaluation framework for fine-grained annotations of long summaries that can be used to evaluate coherent narratives. |
| Outcome: | The proposed framework can support future work in document summarization and coherence evaluation, the authors show . |
CoUDA: Coherence Evaluation via Unified Data Augmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing data augmentations for coherence evaluation rely on heuristic rules and lack designing criteria. |
| Approach: | They propose a data augmentation framework that breaks down coherence into global and local aspects and designs augmentation strategies for both aspects. |
| Outcome: | The proposed framework surpasses recent models in scoring and ranking tasks with 233M parameters. |
Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks (2021.eacl-main)
Copied to clipboard
| Challenge: | Coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. |
| Approach: | They compare models' performance on synthetic sentences with those on retrieval-based dialog. |
| Outcome: | The proposed models perform poorly on synthetic sentences and retrieval-based dialog tasks. |
Joint Modeling of Entities and Discourse Relations for Coherence Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on coherence modeling focuses on entity features or discourse relation features, with little attention given to combining the two. |
| Approach: | They propose two methods for jointly modeling entities and discourse relations for coherence assessment. |
| Outcome: | The proposed methods significantly improve the performance of coherence models on three benchmark datasets. |